1. Field of the Invention
This invention relates to improvements in software and hardware design verification. More particularly, this invention relates to methods and systems for centrally managing the execution of multiple design verification test suites that were written for use by different test harnesses or frameworks.
2. Description of the Related Art
The meanings of acronyms and certain terminology used herein are given in Table 1. The terms Sun, Sun Microsystems, Java, J2EE, J2ME, J2SE, and the Sun logo are trademarks or registered trademarks of Sun Microsystems, Inc., in the United States of America and other countries. All other company and product names may be trademarks of their respective companies. A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.
TABLE 1APIApplication programming interfaceCLDCConnected, limited device configuration. CLDC issuitable for devices with 16/32-bit RISC/CISCmicroprocessors/controllers, having as littleas 160 KB of total memory available.DTDDocument type definitionHTMLHypertext markup languageJAXPJava API for XML ProcessingJ2EEJava 2 Enterprise EditionJ2MEJava 2 Micro EditionJ2SEJava 2 Standard EditionJADJava application descriptorJARJava archiveJTDSJava Device Test Suite Execution FrameworkMIDletA MIDP applicationMIDPMobile information device profile. A set of JavaAPIs, which, together with the CLDC, provides acomplete J2ME application runtime environmenttargeted at mobile information devices.XMLExtensible markup languageXTRFXML test representation format
Tools have been developed in recent years to aid in the design verification of hardware and software systems, for example software suites, hardware circuitry, and programmable logic designs. In order to assure that the design complies with its specifications, it is common to generate a large number of input or instruction sequences to assure that the design operates as intended under a wide variety of circumstances. In general, test systems produce a report indicating whether tests have been passed or failed, and, in some cases may even indicate a module that is estimated to be faulty.
Conventionally, test systems employing complex test suites employ a computer-implemented testing framework for computing devices, such as mobile information devices, and for software designed to run on such devices. A developer submits a computing product under development, typically a computing device or software that is designed to run on the device to the test system, which runs a selected battery of test programs on the product while monitoring its behavior. Each product under test requires an instance of an execution test harness, or the use of a stand-alone test execution API.
In environments where testing of a product is ongoing, different aspects may be tested by different teams. As test results are evaluated, it is often necessary to revise the product under test, or to modify the test suites themselves. In such an environment, communicating such revisions to the different testing teams, maintaining version control, synchronization among the teams, and generally coordinating the testing activities is a difficult management problem. Errors could result in inappropriate testing, thus wasting valuable time and testing resources. Indeed, failure of coordination could result in the release of an inadequately tested product into the marketplace. A related problem when many test suites are being concurrently executed is the effort of setting up each test suite with its own test harness and environment. Bundling the test harness with the test suite is not a good solution, as the effort in maintaining up-to-date versions becomes formidable as the number of concurrently operating test suites increases, and when the product or the test suites are frequently modified by different test teams.